Comparing three Critic Models of Reinforcement Learning in the Basal Ganglia Connected to a Detailed Actor in a S-R Task

Actor-Critic architectures of reinforcement learning were found to show a strong resemblance with known anatomy and function of a part of the vertebrate's brain: the basal ganglia. Based on this analogy, a large number of Actor-Critic models were simulated to reproduce behaviours of rats performing laboratory tasks. However, most of these models were tested in different tasks and it is often difficult to compare their efficiency. The work presented here concerns the comparison of three Critics, tested with the same Actor part taking into account known basal ganglia anatomy. The specificities of the three Critics connected to this Actor lie on the absence or presence of a temporal representation of stimuli, and on the use of one or more units for the prediction of reward. These architectures are implemented in the same simulated robot performing the same stimulus-response (S-R) experiment: a reward-seeking task. Results show that both temporal representation of stimuli and multiple prediction units are mandatory for the achievement of the task. Improvements for Critic modelling and competing hypotheses for Actor-Critic models are finally discussed.